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Abstract 

Background: Despite growth in implementation research, limited scientific attention has focused on understanding 
and improving sustainability of health interventions. Models of sustainability have been evolving to reflect 
challenges in the fit between intervention and context. 

Discussion: We examine the development of concepts of sustainability, and respond to two frequent assumptions — 
'voltage drop,' whereby interventions are expected to yield lower benefits as they move from efficacy to effectiveness to 
implementation and sustainability, and 'program drift,' whereby deviation from manualized protocols is assumed to 
decrease benefit. We posit that these assumptions limit opportunities to improve care, and instead argue for 
understanding the changing context of healthcare to continuously refine and improve interventions as they are sustained. 
Sustainability has evolved from being considered as the endgame of a translational research process to a suggested 
'adaptation phase' that integrates and institutionalizes interventions within local organizational and cultural contexts. 
These recent approaches locate sustainability in the implementation phase of knowledge transfer, but still do not address 
intervention improvement as a central theme. We propose a Dynamic Sustainability Framework that involves: continued 
learning and problem solving, ongoing adaptation of interventions with a primary focus on fit between interventions and 
multi-level contexts, and expectations for ongoing improvement as opposed to diminishing outcomes over time. 

Summary: A Dynamic Sustainability Framework provides a foundation for research, policy and practice that supports 
development and testing of falsifiable hypotheses and continued learning to advance the implementation, transportability 
and impact of health services research. 
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Background 

As implementation science has grown [1,2], researchers 
have advanced from study of facilitators and barriers 
that influence uptake of effective programs and policies 
to investigations of strategies to improve uptake. How- 
ever, often studies evaluate only initial intervention adop- 
tion and implementation. Sustained practice change and 
broader scale-up of interventions [3] rarely are investi- 
gated, often due to the constrained timeframes for research 
that are set by grant mechanisms, and the budgetary and 
political necessity of many decision-makers to take on a 
short-term lens. 

Recently, there has been interest in understanding and 
influencing the sustainability of implemented interventions. 
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While this is progress, frequently used conceptualizations 
of sustainability implicitly replicate assumptions and 
limitations inherent in the traditional research-to-practice 
pathway [4,5], or in its more recent conceptualization 
as translational research [6]. These conceptualizations of 
knowledge translation often assume that interventions are 
optimized prior to implementation, and that they are largely 
independent of the context in which they are delivered [7]. 

The presumed linear model of intervention develop- 
ment, efficacy testing and implementation has resulted 
in the development of an armamentarium of efficacious 
healthcare treatments, preventive strategies, and public 
health interventions. Whfle these discoveries have made 
advances in a number of health domains, they are often 
difficult to implement in a myriad of practice settings 
and even harder to sustain over time and in many real 
world and low- resource settings [8]. In addition, inter- 
ventions are traditionally expected to perform worse in 
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real-world practice than in the laboratory or the rarified 
clinical trial setting. We argue for a new approach to 
sustainability that instead integrates the themes of adap- 
tive, contextually sensitive continuous quality improve- 
ment (CQI) and a learning healthcare system with the 
challenge of intervention sustainment. 
The purposes of this article are to explicate: 

1. Evolving understandings of sustainability and of 
related concepts of CQI and the learning 
healthcare system; 

2. An iterative, dynamic approach to sustainability, 
termed the 'Dynamic Sustainability Framework' (DSF) 
that integrates and extends these concepts; and 

3. Implications of this framework for research, policy, 
and practice. 

Given the variation with which terms central to dis- 
semination and implementation research can be used, 
we include a table that lays out working definitions for 
the central terms of this debate Table 1. 



Moving beyond Voltage drop' and 'program drift 

While the traditional linear process of intervention deve- 
lopment, derived from pharmaceutical medication deve- 
lopment models [13], has often resulted in the creation of 
initially successful interventions, it may be less helpful in 
enabling these innovations to maximally benefit health. A 
linear approach may be particularly challenging to apply 
to complex, multi-component interventions, psychosocial 
treatments, treatment of the growing number of people 
with multimorbid conditions [14,15], and systemic ap- 
proaches to care [15]. Linear approaches place a premium 
on creating and 'freezing' an intervention, developing 

Table 1 Definitions of key terms used in this paper 



Term 



Definition 



Implementation The process of putting to use or 

integrating evidence-based interventions 
within a setting [9]. 

Sustainability To what extent an evidence-based 

intervention can deliver its intended 
benefits over an extended period of time 
after external support from the donor 
agency is terminated [9]. 

Sustainment The continued use of an intervention 

within practice [10]. 

Voltage drop The phenomenon in which interventions are 

expected to yield lower benefits as they move 
from efficacy to effectiveness and into 
real world use (adapted from [1 1]). 

Program drift The phenomenon whereby deviation from 

manualized protocols in real-world delivery of 
interventions is expected to yield decreasing 
benefit for patients (adapted from [1 2]). 



manuals to ensure its consistent delivery with fidelity [16], 
and then minimizing deviations from the intervention. For 
example, the COMMIT stop smoking national project had 
expert researchers design the complex, lengthy interven- 
tion protocol based on research evidence, and then gave a 
several hundred page manual of operations to local staff 
to implement in their communities [17]. 

The importance of internal validity to the scientific pro- 
cess should not be ignored, but its overemphasis relative to 
generalizability and adaptation runs the risk of creating 
interventions that will not fit within different, complex or 
changing settings and of failing to benefit settings, clini- 
cians, and patient populations who are underrepresented in 
the intervention testing process [7,16]. Two key implicit 
assumptions within the traditional intervention develop- 
ment approach may limit ultimate progress toward inter- 
vention sustainability and population impact: 

First, interventions are often developed with the idea 
that they can be optimally constructed, manualized, and 
then tested in a single form applicable across settings 
and over time. Efficacy trials are designed to screen out 
noise in the system (patient comorbidities, competing de- 
mands and skill variance of clinicians, resource limitations, 
varying motivations of patients) [7], and thus maximize out- 
comes. As interventions move to effectiveness and into 
implementation, one expects that the individual benefit of 
the intervention will likely drop, due to the added complex- 
ity of heterogeneous patients, providers and settings. This is 
referred to as Voltage drop' (Figure 1). The assumption of 
Voltage drop' results in missed opportunities to refine and 
improve the intervention, instead concluding that the de- 
clining benefit is expected and acceptable, and the best pos- 
sible outcome is that which is achieved at the efficacy stage. 

Second, the assumption that interventions can be opti- 
mally constructed in the early stages of the development 
and testing process, independent of context, suggests 
that, even at the stages of implementation and sustain- 
ability, change to the intervention is expected to have 
negative consequences, and that the further a practi- 
tioner deviates from the manual, the lower the benefit. 
This is the concept of program drift' (Figure 1). Delive- 
ring the intervention within an efficacy trial may require 
adherence to protocols that are challenging to deliver 
within real-world practice. Fidelity ratings then assume 
that 100% fidelity to original protocols will yield optimal 
outcomes, and effort is expended to ensure that practi- 
tioners do not deviate from the manual. Where clini- 
cians do deviate from the protocol, the field expects that 
the resulting program drift' will compromise outcomes 
[12]. We see that this over-reliance on quality assurance 
to prevent program drift' leads to extensive pressure on 
real-world practices to adhere to the intervention proto- 
cols without evidence that this adherence will lead to 
optimal outcomes. Quality assurance may inadvertently 
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Figure 1 Program drift and voltage drop. Illustrating the concepts of 'program drift,' in which the expected effect of an intervention is presumed 
to decrease over time as practitioners adapt the delivery of the intervention (A), and 'voltage drop,' in which the effect of an intervention is presumed 
to decrease as testing moves from Efficacy to Effectiveness to Dissemination and Implementation (D&l) research stages (B). 



hamper sustainability and ongoing improvement, cus- 
tomization and optimization of interventions to the det- 
riment of population health. 

In contrast, we reject the notion that an intervention can 
be optimized prior to implementation, and explicitly reject 
the validity of program drift' and Voltage drop.' Rather, we 
suggest that the most compelling evidence on the maximal 
benefit of any intervention can only be realized through 
ongoing development, evaluation and refinement in diverse 
populations and systems [18]. Instead of viewing contextual 
factors as interfering with the delivery of an effective inter- 
vention and needing to be controlled, we see the opportu- 
nity to learn about the optimal fit of an intervention to 
different care settings [2]. For example, strategies have been 
developed to adjust organizational characteristics {e.g,, cul- 
ture, climate, structure) to enable improved fit between the 
intervention and the setting [19]; harnessing the under- 
standing of context can enable beneficial adaptation of the 
intervention and improve sustainability. Without rejecting 
these assumptions, we reify early phase interventions tested 
in the most artificial settings, set quality assurance of inter- 
ventions as an optimal outcome, and miss opportunities 
for continued learning and development. 

Understanding and advancing sustainability research 

As the field of implementation science has matured [20,21], 
more emphasis has been placed on understanding sustain- 
ability. Researchers have recognized that implementation 
of interventions, which can often require substantial re- 
sources, is meaningless without successful long-term use. 
Following Rabin et al/s glossary for Dissemination and 
Implementation Research in Health [9], we draw dis- 
tinctions between 'implementation/ which relates to the 



initial process of embedding interventions within set- 
tings and sustainability,' which relates to the extent that 
these interventions can continue to be delivered over time, 
institutionalized within settings, and have necessary ca- 
pacity built to support their delivery. 

Recent articles [22-24] have advanced the idea of an 
adaptation phase that bridges from the initial implemen- 
tation effort to a longer-term sustainability phase. They 
argue the need to examine the fit between the practice 
setting and the intervention and make changes necessary 
to improve the integration of the intervention into on- 
going care processes. This is consistent with the institu- 
tional theory of organizations, which argues that the final 
stage of innovation requires the 'institutionalizing' of the 
new practice so that it becomes a working part of the 
organization [25]. 

As a consequence, assessment of organizational cha- 
racteristics {e.g., structure, climate, culture, resources) is 
seen as an essential component of sustainability, and in- 
deed, the fit between context and the intervention is at 
the center of a sustainability phase [24]. There has also 
been an emphasis on planning for sustainability much 
earlier in the intervention process [26]. Recent approaches 
to sustainability locate key efforts squarely in the imple- 
mentation phase, arguing that once a practice has been 
implemented within a care system, those who manage the 
delivery of that practice should turn their attention to 
ensuring that the practice can be maintained over time 
[6,24] . Authors typically suggest that this entails attention 
to issues of long-term financing, training of the workforce, 
supervision, and organizational support for the practice 
[26,27]. A characteristic of this approach is to postpone 
emphasis on sustainability until after implementation is 
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well underway, assuming that implementation and sus- 
tainability are sufficiently independent. 

Authors have also highlighted the utility of assessing 
outcomes of those who have received the practice, some- 
thing infrequently collected in routine practice [24,28]. 
Seldom is it demonstrated that continued delivery of an 
intervention confers benefit on the patient population 
that receives the intervention or the system that delivers 
it {e.g,, cost containment, efficiency of care, quality met- 
rics). Measurement of outcomes over time to determine 
continued benefit has been shown to support sustain- 
ability of the practice [29,30]. 

Recent implementation projects have created new 
tools and scales to study sustainability, including needs 
assessments, long-term action plans, tracking of pro- 
gram adaptation, financial planning, mapping of com- 
munity networks, and measurement of the degree to 
which practices are integrated and institutionalized into 
service systems [31-33]. While this emerging focus on 
sustainability is an advance, many studies still assume a 
largely static service delivery system that needs to be 
assessed only at key time points, but not in an ongoing 
manner. To better reflect complexity and change within 
the system and in context, a more dynamic approach to 
sustainability is needed. 

Framework 

The dynamic sustainability framework (DSF) 

As Heraclitus observed, 'The only constant is change.' 
The Dynamic Sustainability Framework has developed 
out of our evolving thinking and our collective experience 
in conducting and advancing implementation science. 



where we have seen inattention to constant change limit 
the ability to which implemented interventions are sus- 
tained over time in complex clinical and community set- 
tings. The DSF (Figure 2) emphasizes that change exists in 
the use of interventions over time, the characteristics of 
practice settings, and the broader system that establishes 
the context for how care is delivered. As classical thinking 
eloquently captures, change impacts the ability of health 
interventions to be optimally used and sustained over 
time. This dynamism exists in the evidence base for inter- 
ventions that links causal factors to health outcomes, as 
judged by the continual stream of new publications in aca- 
demic journals that add to available evidence on the ef- 
fectiveness of interventions, as well as ongoing practice 
surveillance systems that capture intervention impact. 
Dynamism exists in the interventions that support the evi- 
dence, which acknowledge ad hoc adaptation and experi- 
mentation of evidence-based interventions. Furthermore, 
it exists in a constantly changing multi-level context [34], 
internal to a clinical or community setting and the broader 
care system, be it an organization, community, county, 
state or country. 

The DSF, like many implementation models, centers 
on a few major elements: the intervention, the context 
in which the intervention is delivered, and the broader 
ecological system within which the practice settings exist 
and operate. Distinct from those models, however, is the 
consideration of these elements over time. The interven- 
tion, as shown in the figure, often includes a set of in- 
dividual components chosen for their ability to effect 
behavior or biochemical change, an assumed set of cha- 
racteristics defining who should deliver the intervention. 




Figure 2 The dynamic sustainability frameworlc. Illustrating the goal of maximizing the fit between interventions, practice settings, and the 
broader ecological system over time (represented by To, Ti Tp), each of which has constituent components that may vary. 
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targeted, patient-centered outcomes that the interven- 
tion should generate as a result of its use, and a delivery 
platform {e.g,, face-to-face, telephonic, web-based, mo- 
bile health app, etc.). Other constructs may also define 
the intervention. 

The DSF anchors the ultimate benefit of the interven- 
tion in terms of its ability to fit within a practice setting, 
typically a clinical or community setting. This context car- 
ries its own set of characteristics, including human and 
capital resources, information systems, organizational cul- 
ture, climate and structure, and processes for training 
and supervision of staff. The DSF, consistent with other 
models, argues that these practice characteristics will 
directly influence the ability of the intervention to reach 
the patient population that could benefit, and thus mea- 
surement of these contextual constructs is paramount to 
resolving fit. 

At a third level, the DSF identifies the ecological system 
as an additional driver of the successful implementation 
and sustainability of an intervention. The ecological sys- 
tem consists of many practice settings that influence those 
working to incorporate a particular intervention, as well 
as the legislative and regulatory environment, characteris- 
tics of local, regional, state and national markets, and 
characteristics of the broad population. The ecological sys- 
tem is influenced by changes to available interventions 
and practice settings, and in turn, influences them. 

Specific to the DSF, as emphasized by the dotted lines 
in Figure 2, is the expectation that change is constant at 
each of these levels (and ripples across multiple levels), 
and thus the success of an intervention to be sustained 
over time lies in the measured, negotiated, and recipro- 
cal fit of an intervention within a practice setting and 
the practice setting within the larger ecological system. 
The DSF suggests that optimal fit requires that charac- 
teristics of the intervention, practice setting, and eco- 
logical system be consistently tracked, using valid, reliable 
and relevant measures, and expects that interventions, 
settings and the ecological system should change over 
time, particularly where data can suggest improve- 
ments for each to better meet the needs of patients, 
the skills and resources within the practice setting, and 
the larger ecology. 

The DSF is intended to suggest a new paradigm to 
consider the long-term use and ongoing improvement of 
interventions, recognizing the limitations of the evidence 
base made available through efficacy and effectiveness trials, 
and allowing that continuous exposure of the intervention 
to new populations, new contexts, and new innovations 
can result in continued improvement of resulting out- 
comes, thus minimizing the perils of program drift' and 
Voltage drop.' Indeed, the DSF posits that ongoing quality 
improvement of interventions is the ultimate aim, not qua- 
lity assurance of them. To be clear, we see value in ensuring 



appropriate quality assurance within healthcare systems 
where clear assessments of an appropriate standard of care 
are made through knowledge of core intervention com- 
ponents. However, the DSF recognizes the limitations of 
intervention evidence solely from clinical trials and argues 
that quality improvement processes focused on interven- 
tion optimization are ultimately more relevant to achieve 
sustainment. 

The DSF, which has benefitted from the authors' on- 
going dialogue with the Implementation Science com- 
munity about the challenge of sustainability, follows the 
spirit of a number of existing models that emphasize 
three things — importance of context, the need for on- 
going evaluation and decision-making, and the goal of 
continuous improvement. These include Wandersman's 
Getting to Outcomes model [31], Continuous Quality 
Improvement (CQI) [34], system dynamics [35], com- 
plexity theory [36], adaptive management [37], and the 
Evidence Integration Triangle [30]. In addition, the DSF 
is consistent with alternative views of organizational 
development [38] and the principles of system science 
[39]. Distinct in the DSF from many of these other 
models is the emphasis on omnipresent change, and the 
central goal of continuously optimizing the fit between 
the intervention and a dynamic delivery context to achieve 
maximal benefit. The DSF is anchored around the follo- 
wing seven tenets, for which we think there is evidence, 
but recommend explicit testing in this context: 

An intervention should not be optimized prior to 
implementation, or even prior to 'sustainability 
phase' onset 

Interventions benefit from ongoing optimization as they 
are applied in different contexts [37]. The evidence that 
supports the benefits of health interventions arises from 
trials that represent a very small slice of the diversity of 
demographics, preferences, and health status of the 
population at large [7,40], but we should not expect evi- 
dence collected in one set of narrow, relatively optimal 
circumstances to apply perfectly in other, vastly different 
contexts [41]. A corollary' of this recommendation is 
that, other things being equal, quality improvement ap- 
proaches that involve adjusting and refining program 
should be more effective than quality assurance' proce- 
dures that emphasize fidelity to an initial protocol. 

Interventions can be continually improved, boosting 
sustainment in practice, and can enable ongoing learning 
among developers, interventionists, researchers 
and patients 

There is tremendous opportunity to aggregate evidence 
on the real-world impact of interventions when used in 
practice. We can apply models of continual refinement 
that have been the cornerstone of software development 
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{e.g., Firefox, Reaper, iTunes), as well as Web 2.0 sites {e.g., 
Wikipedia, Facebook, etc.) [42]. By augmenting trial data 
with practice-based evidence, we can understand much 
more about what works for whom, the question under- 
lying personalized medicine [43,44]. This articulation of 
the DSF suggests the need for a long-term plan to commit 
resources for training and ongoing improvement. One im- 
plication of the DSF is that intervention impact can also 
be enhanced through increases in efficiency. The field has 
developed a plethora of multi-component interventions, 
often without studies that determine what the minimal set 
of components are needed to ensure benefit [45]. The 
DSF, congruent with the Consolidated Framework for 
Implementation Research [46], emphasizes the impor- 
tance of streamlining interventions to peel away compo- 
nents that may not be central to improving outcomes or 
to adapt intervention components to a particular context. 

Ongoing feedback on interventions should use practical, 
relevant measures of progress and relevance 

Too often, intervention trials focus on markers that are 
psychometrically valid but of less relevance to patients and 
clinicians [3,47]. For example, very specific, intervention- 
related symptomatic scales may be most sensitive to 
change, but there is little guarantee that the measured 
change translates into a tangible, ftinctional benefit for the 
patient [48]. The DSF thus suggests the use of measures 
such as checklists that are relevant to desired outcomes of 
patients, as well as sensitive to the Tit' between interven- 
tions and context, and can be feasibly implemented [49]. 
Across each of the changes in Figure 2, we see available 
streams of data that can offer leverage points for improving 
interventions. Environmental changes, for example, can be 
tracked via population surveys, and market and claims data. 
Practice changes can be captured through electronic health 
records, claims data and practice surveys. Evidence reviews 
can provide key information on knowledge changes, and 
policy changes can be tracked via available Federal and 
Non-Profit sources {e.g., CMS, Kaiser Family Foundation, 
etc.). This is consistent with the CQI model, but with 
specific emphasis on the ongoing refinement of the inter- 
vention to counteract the assumptions of 'Program Drift.' 

Voltage drop is NOT inevitable 

We reject the assumption that the more diverse and 
complex a patient population is, the smaller the benefit 
of intervention, referred to above as Voltage drop' [11]. 
This stems from an expectation that intervention studies 
require control of the environment to isolate a treatment 
effect [7]. If we embrace CQI of the specific health inter- 
vention, we expect that with more experience, we will 
better be able to adapt interventions to contexts and pa- 
tients. As we learn more about what works and what 
doesn't and adjust protocols accordingly, the Voltage' 



could maintain or possibly increase over time. This echoes 
the computer industry, where each new release of a hard- 
ware or software line is expected to be better than the 
prior version. It also finds consonance with the evolution 
of the flu vaccine, which is constantly refined in response 
to the changing nature of the influenza virus each season. 
A culture of improvement is central to ongoing interven- 
tion use and treats improvement of the intervention as 
central to the sustainability process [33] . 

Programs should be more likely to be maintained when 
there is strong 'fit' between the program and the 
implementation setting 

The concept of Tit' has been discussed by other authors 
{e.g., Estabrooks, Glasgow, Dzewaltowski), [50] and goes 
back at least as far as Rogers' Diffusion of Innovations 
[51], where the concept of reinvention evoked the notion 
of departing from the original intervention concept to 
create' a new version suited to the preferences and con- 
straints of the local context. Fit is a multi-level construct 
and involves alignment along multiple dimensions [50]. 
The DSF posits that fit will likely change over time, due 
to changes in the way in which an intervention is deli- 
vered, the characteristics of patients, providers and set- 
tings, and the broader ecological system within which 
healthcare settings reside. Attention to this fit, through 
ongoing assessment and quality improvement efforts, 
should improve sustainment and ultimately identify oppor- 
tunities for intervention improvement. 

Organizational learning should be a core value of the 
implementation setting 

While training and ongoing analyses exist in many organi- 
zations, the demands imposed by multiple levels of change 
require learning to be central to organizational activity for 
interventions to become sustainable. The context both 
within an organization and in the broader ecological sys- 
tem is constantly changing, and requires a learning orga- 
nization' [52] to engage in problem-solving capacity at 
multiple levels [53]. Organizational learning should also 
target appropriate adaptation of evidence-based interven- 
tions, possibly in rapid learning cycles [44,54], followed by 
ongoing assessment and feedback loops. The DSF is con- 
gruent with the concepts of the learning healthcare system, 
again with the emphasis on learning how to better develop, 
deliver and sustain interventions. 

Ongoing stakeholder involvement throughout should 
lead to better sustainability 

Continuously engaging stakeholders throughout the plan- 
ning, implementation and adaption processes should help 
increase the fit between the intervention and the local 
context, and help address evolving issues that might inter- 
fere with sustainability. Just as researchers have proposed 
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intervention development processes [12] that focus on 
the ultimate site where interventions will be delivered, we 
argue that partnership among all relevant stakeholders is 
essential to maintaining and improving interventions within 
care settings. 

As Figure 3 depicts, we view sustainability as akin to 
the challenge of fitting a puzzle piece within an evolving 
large tableau. Without sensitivity to the characteristics 
of the intervention, practice setting and the larger sys- 
tem, there is little expectation that the intervention will 
fit well within the setting, and as the context changes 
(noted by the changing shape in the figure), sustainment 
will be harder and harder to achieve. However, by exa- 
mining and adapting the intervention to a changing con- 
text, we believe that sustainment is not only possible, 
but that the utility of the intervention can be optimized. 
The experience of delivering the intervention in vivo 
over time serves to inform the ongoing evolution of the 
intervention (noted by the change in shape of the puzzle 
piece). By concentrating on the dynamic 'fit' between an 
intervention and its delivery context as the core ingre- 
dient underlying sustainability, we embrace opportu- 
nities to refine and improve the intervention. 

Contrasting static and dynamic views of sustainability 

Table 2 compares static views of sustainability with the 
DSF, offering sharp contrasts in data collection and 



analysis, opportunities for knowledge development and in- 
corporation of the noise' within healthcare contexts. Ra- 
ther than seeking to simplify the phenomenon of study, 
either by avoiding adaptation of interventions, or assuming 
the context to be unchanging, the DSF embraces change as 
a central influence on sustainability. Adaptation is ex- 
pected, and even encouraged. Assessment of care settings 
and outcomes is ongoing and incorporated within practice, 
and staffing and policy changes are incorporated in sus- 
tainability planning. Perhaps the biggest contrast of the 
static and dynamic views is that the static view limits 
lessons that can in turn provide feedback to other areas of 
science; the DSF views an abundance of ongoing evidence 
that can be cycled to continuously improve intervention 
design, testing, and ongoing system change. 

Discussion 

Implications of the DSF 

This initial formulation of the DSF has implications for 
future practice, research and policy. For practice, the 
DSF highlights the need for continuous assessment of 
the local context, not just prior to implementation. This 
enables care settings to better manage the fit between 
their resources, needs and the interventions, including 
generating consistent feedback on how interventions are 
delivered to diverse patients and how patients do as a 
result. The coUection and analysis of this information 




Figure 3 Using the dynamic sustainability framework as an engine for quality improvement. The DSF depicts a dynamic view of 
sustainability, wliicli allows for the evolution of an intervention within a changing delivery system. The changes in the shape of the puzzle pieces 
and of the contexts reflects the ongoing change to interventions, practice settings, and care systems, and shows the use of quality improvement 
methods to optimize the 'fit' and improve the public health benefit of sustained use of interventions. 

V J 
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Table 2 Contrasting static views of sustainability with the dynamic sustainability framework 



Static view Dynamic sustainability view 



Adaptation 


Bad; avoided/eliminated 


Inevitable; encouraged, monitored and guided by evidence 


Context assessment 


Initial or during implementation 


Ongoing 


Outcomes assessment 


During study by researchers 


Incorporated as part of organization 


Review of evidence 


Initial- from efficacy studies 


Ongoing; from convergent sources including replications 


Staffing issues {e.g., tumover) and variations 


Ignored/feared 


Planned for; investigated 


Generates new l<nowledge 


No 


Yes, feedback to other areas of science and to earlier stages 



Note. This table contrasts more traditional static views of sustainability, In which efforts are made to minimize change and retain the original form of an 
intervention, from a dynamic sustainability view that we suggest In the DSF, in which change is inevitable and can lead to better fit and ultimately better impact 
of Interventions. 



allows practitioners to make informed decisions about 
how best to utilize existing interventions, allows for po- 
tential enhancements to the interventions to be made 
and shared, and offers better information on which to 
make decisions to cease delivering interventions that do 
not have benefit The intention is to recognize and support 
rapid learning, real-time problem-solving organizations [54] 
that are full partners in the generation of knowledge, not 
just its application. Thus, the DSF promotes the use of mul- 
tiple methods of planning for sustainability, including simu- 
lation modeling of the impact of different decisions, pilot 
testing of adaptations within local contexts, and continued 
experimentation. Perhaps an even greater benefit to prac- 
tice would come through pooling of data across a larger set 
of sites, practitioners and patients, something done with 
success for chronic disease [55,56]. 

For research, the DSF dispels the notion that interven- 
tion development, refinement and improvement are com- 
pleted prior to real world implementation. In contrast, 
we suggest that development and refinement is never 
complete. Rather, sustainability is the process of managing 
and supporting the evolution of an intervention within a 
changing context. We recommend (and welcome testing 
of the idea) that programs that monitor context and adjust 
accordingly do better long-term. In addition, we see re- 
search studies testing whether settings and programs 
using ongoing CQI or other means of feedback improve- 
ment perform better over time. More broadly, we see the 
DSF as changing the notion of a linear transition from 
research to practice into a shared process of continual ex- 
perimentation and analysis through the use of both prac- 
tice settings and ecological systems to track changes and 
assess evolving fit between interventions and practice set- 
tings. Principles of crowd sourcing' made popular within 
the IT industry and resulting in open source products like 
Firefox, Wikipedia and Reaper, blur the lines between re- 
search and practice. We see this in the evolution of the 
electronic health record, and the rapidly evolving patient 
health record, which is dramatically changing the scenario 
from one where the medical expert makes all the decisions 
to one of collaborative care and shared decision-making 



[57,58]. Instead of a small team of researchers developing 
a priori an optimal,' static product, a large and often vir- 
tual community including users and consumers continu- 
ously upgrades dynamic products. 

This initial formulation of the DSF also has implications 
for policy. Incentives are needed to support ongoing adap- 
tation of interventions, particularly where evidence is li- 
mited, specifically including monitoring of progress and 
documentation of adaptations, using quality measures rele- 
vant to stakeholders and patients. In addition, research 
funders must determine how to support longer-term pro- 
jects related to sustainability with flexible research designs, 
since the ultimate benefit of integrating and modifying 
interventions may not be evident for many years. In 
addition, infrastructure to support pooling of 'practice- 
based evidence' will be needed, in order to ensure suffi- 
cient information is available about long-term use and 
adaptation of interventions. The DSF aligns directly with 
a number of existing policy initiatives, at national, state 
and local levels, including the advance of Patient- Centered 
Medical Homes, Accountable Care Organizations, Pay for 
Performance initiatives, and support for local demonstra- 
tion projects. 

We recognize that this conceptualization of the DSF 
model should, consistent with its internal logic, be refined 
and improved over time. Whether this happens through 
testing of the tenets laid out in the previous section or 
contributions of others' theoretical or empirical studies, 
we offer the DSF as the beginning of a longer debate. For 
example, while Figure 2 shows three levels (intervention, 
practice setting, ecological system), we appreciate that 
many more levels of the system exist than what we have 
depicted in the figure. We see further specification of the 
interrelationships of those levels as a useful area for fur- 
ther research and development. In addition, we see the 
utility of aligning the DSF with alternative methods of de- 
veloping and testing dissemination and implementation 
interventions from the prevailing linear model. In an effort 
to present examples of how the DSF might be used to 
consider the sustainability of various types of interven- 
tions, we have included Table 3. 
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Table 3 Illustrative examples of the use of DSF for different types of interventions 



Intervention example 



Applying DSF principles 



Clinical guidelines for pharmacotherapy 

• Clinical guidelines for pharmacotherapy to treat a range of chronic 
diseases have been developed, implemented and refined as new 
compounds have reached the market and new evidence has been 
gathered about the relative benefit of different medications. 

• The influences on prescribing practices exist at the patient level 
(preferences and predictors of response), clinician level (practice patterns, 
level of training), system level (formulary design, insurance coverage, 
adherence monitoring). 

• Each influence will impact guideline implementation and overall benefit 
of care for patients served within the health system. 



• Assessing appropriate fit between guidelines and the care setting will 
require analysis of multiple streams of data, including administrative, 
clinical, organizational and epidemiologic. 

• The DSF suggests that collecting benchmarks over time on patient 
outcomes, adaptations in algorithms used, available evidence (from the 
literature, healthcare systems and patient populations), and contextual 
factors could result in improvements to the guidelines, to the capacity of 
the health system to more seamlessly integrate the guidelines, to the 
ecological system that could improve access, quality and health outcomes. 



Psychotherapy for mood disorders 



• Manualized evidence-based psychotherapies for mood disorders have 
been tested in numerous studies. 

• Many of these therapies are designed to be delivered by specific 
providers, over a set number of sessions, with a clear step-by-step 
approach. 

• Given variation in access to therapy {e.g., number of sessions covered, 
availability of therapists, time), limited predictive ability of response (how 
many sessions are needed, what are active ingredients, who should 
deliver therapy?), emergent options for mode of delivery (web-based, 
face-to-face, self-guided, asynchronous), optimizing psychotherapy for 
individuals and systems is still beyond our current knowledge base. 



Care management for chronic diseases 

• Studies have shown the effectiveness of care management strategies to 
assess, intervene and monitor for a range of chronic conditions. 

• Typical strategies involve initial screening, assessment, treatment 
planning, care and self-management strategies, and follow-up. 

• While general care management approaches seem to be durable, 
specific approaches can have difficulty being implemented across many 
clinical and community settings, because of limitations in resources (both 
monetary and staffing), information systems, financing processes, and 
other barriers. 

• With new technologies, additional evidence about treatment and 
preventive interventions, and reconfiguration of care systems, care 
management for chronic diseases can be sustained and improved 
in a large variety of care settings. 



The DSF suggests manualized psychotherapy could be improved by 
tracking variation in use and therapeutic response of patients, contextual 
characteristics that influence delivery, and additional interventions that 
affect clinical and functional outcomes. 

Systems could track how patients respond to varying doses of therapy, 
modes of delivery, and clinician characteristics. Over time, decision-makers 
could align available care to the needs of patient populations, and 
clinicians could adapt practice patterns to data on patient preferences 
and outcomes, and general needs of the patient population. 

By assessing the fit of psychotherapy delivery with patients, the service 
setting and the broader ecological system, the DSF hypothesizes that new 
insights about psychotherapy optimization could drive improvements in 
patient care. 



Care management is influenced by drivers at patient, provider, 
organization and system levels. 

Care management requires coordination among multiple people, 
organizational supports and capital resources, all of which will likely shift 
over time. 

Therefore, care management cannot be sustained without continual 
assessment of fit within the local setting and the support of the 
components of the model. 

The DSF hypothesizes that attention to local adaptations made by 
healthcare and community settings to fit the model; and feedback on 
staffing levels, intensity of care management, emerging interventions and 
patient outcomes could enhance long-term sustainability and model 
improvement. 

Evidence about who benefits most from different variants of care 
management, who is an ideal care manager, and what are the best ways 
of coordinating across primary care and specialty practice could lead to 
better uptake and improve patient health. 



Note. This table offers examples of how the sustainability of different types of interventions can be enhanced by applying the principles of the DSF. Both the 
descriptions of the Interventions and the use of the DSF to improve them are exemplars to further the debate, and are not intended to comprehensively depict 
the extensive levels of Influence. 



Summary 

It is time to embrace the culture of a learning healthcare 
system [44,59] to promote sustainability of interventions 
that are optimized and customized to the myriad of 
clinical and community settings. The enormous changes 
in health systems in the past few years give particular 
salience to a conceptualization of sustainability as a 



dynamic process, and provide unparalleled opportunity to 
test and refine the principles offered in this paper. With- 
out this emphasis, we reify the past by asserting the pri- 
macy of evidence-based guidelines based on evidence 
from rarified clinical trial settings, and give second-class 
status to ongoing learning in real world settings. Allowing 
researchers to be the only ones allowed to generate new 
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knowledge limits the opportunities to consistently im- 
prove the care we provide. In the past, these limitations 
were technological; we lacked the tools and the sources of 
information to drive rapid improvements. In this era of 
crowd sourcing,' of exponentially-expanding processing 
power and global connectedness, we no longer need to 
adhere to a view that once created, interventions and 
healthcare settings must be 'frozen' to optimize effective- 
ness. Instead, we propose the DSF as helping to reconfig- 
ure the research-practice-policy interface, in which the 
best possible information is gathered and used in real- 
time to inform policy, improve practice, and answer the 
highest priority research questions. Only then will the 
promise of a learning healthcare system be reached. 
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